SMSTC (2007/08)

Probability

                www.smstc.ac.uk

Contents

5 Important special distributions 5–1 5.1 Binomial, Poisson, and multinomial . . . . . . . . . . . . . . . . . . . . . . . . . . 5–1 5.2 Thinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–3 5.3 Geometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–4 5.4 Uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–5 5.5 Exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–6 5.6 Gaussian (normal) variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–8

5.6.1 The Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–8 5.6.2 The multidimensional Gaussian random vector . . . . . . . . . . . . . . . 5–9 5.6.3 Conditional Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–12

SMSTC (2007/08) Probability

Lecture 5: Important special distributions

a

                  www.smstc.ac.uk

\underline{Contents}

5.1 Binomial, Poisson, and multinomial . . . . . . . . . . . . . . . . . . . 5–1 5.2 Thinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–3 5.3 Geometric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–4 5.4 Uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–5 5.5 Exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–6 5.6 Gaussian (normal) variables . . . . . . . . . . . . . . . . . . . . . . . . 5–8 5.6.1 The Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5–8 5.6.2 The multidimensional Gaussian random vector . . . . . . . . . . . . . . 5–9 5.6.3 Conditional Gaussian law . . . . . . . . . . . . . . . . . . . . . . . . . . 5–12

5.1 Binomial, Poisson, and multinomial

Consider the coin tossing experiment, i.e. consider a sequence ξ_{1}, ξ_{2}, … of i.i.d. random variables with

P(ξ_{1} = 1) = p, P(ξ_{1} = 0) = 1 − p.

The law of

S_{n} = ξ_{1} + ⋯ + ξ_{n}

is called Binomial with parameters n and p. From this we have

ES_{n} = np, var S_{n} = n var ξ_{1} = np(1 − p),

and, since ξ_{n+1} + ξ_{n+m} is Binomial with parameters m and p, and independent of S_{n}, we have that the sum of two independent Binomial random variables with the same p is again Binomial. We also have

P(S_{n} = k) = \binom{n}{k}p^{k}(1 − p)^{n−k}, k = 0, 1, …, n,

and, from the Binomial theorem,

Ee^{θS_{n}} = (pe^{θ} + 1 − p)^{n}.

EXERCISE 1. Compute the moment generating function of n^{−1/2}(S_{n} − ES_{n}) and show that, as n → ∞, it converges to the a function of the form e^{−cθ^{2}}. (This is moment generating function of a Gaussian random variable).

EXERCISE 2. Use Chernoff’s inequality to estimate P(S_{n} > n(p + x)) for x > 0.

SMST C: Probability 5–2

5–1 By letting p vary with n and taking limits we obtain a different fundamental law, the Poisson law. Specifically,

Lemma 5.1. If p_{n} = \frac{λ}{n} + o(1/n), as n → ∞, then

P(S_{n} = k) → \frac{λ^{k}}{k!} e^{−λ},

for all k = 0, 1, 2, . . ..

Proof Use Stirling’s formula. □

This is the Poisson law with parameter λ. The Poisson law is fundamental when, roughly speaking, we deal with independent rare events. Let X be a Poisson random variable. Then We have

Ee^{θX} = ∑_{k=0}^{∞} \frac{(λe^{θ})^{k}}{k!} e^{−λ} = e^{λ(e^{θ}−1)}.

Differentiating a couple of times, we find

EX = λ, var X = λ.

Furthermore,

Lemma 5.2. If X_{1}, X_{2}, … are independent Poisson random variables with parameters λ_{1}, λ_{2}, … such that ∑ λ_{k} < ∞ then ∑_{k} X_{k} is Poisson with parameter ∑_{k} λ_{k}.

Proof We prove this for a finite number n of random variables (which is enough by the way we construct a probability on an infinite product). The moment generating function of X_{1}+…+X_{n} is

∏_{k=1}^{n} e^{λ_{k}(e^{θ}−1)} = e^{(e^{θ}−1) ∑_{k=1}^{n} λ_{k}}

and this is the moment generating function of a Poisson law with parameter ∑_{k} λ_{k}. □

Next we consider conditional probabilities:

Lemma 5.3. Suppose that X_{1}, X_{2} are independent Poisson with parameters λ_{1}, λ_{2}. Then, conditional on X_{1} + X_{2} = n, we have that X_{1} is Binomial with parameters n, λ_{1}/(λ_{1} + λ_{2}).

Proof Elementary conditioning: For 0 ≤ k ≤ n,

P(X_{1} = k|X_{1} + X_{2} = n) = \frac{P(X_{1} = k, X_{2} = n − k)}{P(X_{1} + X_{2} = n)}

= \frac{\frac{λ_{1}^{k}}{k!} e^{−λ_{1}} \frac{λ_{2}^{n−k}}{(n − k)!}e^{−λ_{2}}}{\frac{(λ_{1} + λ_{2})^{n}}{n!} e^{−λ_{1}−λ_{2}}}

= \binom{n}{k} (\frac{λ_{1}}{λ_{1} + λ_{2}} )^{k} (\frac{λ_{2}}{λ_{1} + λ_{2}} )^{n−k} .

□

Generalising this, we have:

Lemma 5.4. Suppose that X_{1}, X_{2}, …, X_{d} are independent Poisson with parameters λ_{1}, λ_{2}, …, λ_{d}, respectively. Then, conditionally on ∑_{k} X_{k} = n, the random vector (X_{1}, …, X_{d}) has law given by

P(X_{1} = n_{1}, …, X_{d} = n_{d} | ∑_{k} X_{k} = n) = \binom{n}{n_{1}, …, n_{d}} (\frac{λ_{1}}{λ} )^{n_{1}} ⋯ (\frac{λ_{d}}{λ} )^{n_{d}} ,

where (n_{1}, …, n_{d}) are nonnegative integers with sum equal to n, and λ = ∑_{k} λ_{k}, and where

\binom{n}{n_{1}, …, n_{d}} = \frac{n!}{n_{1}! ⋯ n_{d}!}

EXERCISE 3. Show this.

The symbol \binom{n}{n_{1}, …, n_{d}} is the multinomial coefficient since it appears in the algebraic identity

known as multinomial theorem:

(x_{1} + ⋯ + x_{d})^{n} = ∑ \binom{n}{n_{1}, …, n_{d}}x_{1}^{n_{1}} ⋯ x_{d}^{n_{d}}.

The sum is taken over all nonnegative integers (n_{1}, …, n_{d}), with sum equal to n. The random variable (X_{1}, …, X_{d}) with values in ℤ_{+}^{d} is said to have a multinomial distribution with parameters d, n, p_{1}, …, p_{d} (where p_{1} + ⋯ + p_{d} = 1, so one of them is superfluous) if

P(X_{1} = n_{1}, …, X_{d} = n_{d}) = \binom{n}{n_{1}, …, n_{d}}p_{1}^{n_{1}} ⋯ p_{d}^{n_{d}},

where (n_{1}, …, n_{d}) are nonnegative integers with sum equal to n, Of course, a multinomial distribution with parameters 2, n, p, 1 − p is a binomial distribution with parameters n, p.

5.2 Thinning

Suppose an urn contains n balls. There are d colours available. Let the colours be denoted by c_{1}, c_{2}, …, c_{d}. To each ball assign colour c_{i} with probability p_{i}, independently from ball to ball. Let S_{n}^{i} be the number of balls that have colour c_{i}, 1 ≤ i ≤ d. It is easy to see that (S_{n}^{1}, …, S_{n}^{d}) has a multinomial law:

P(S_{n}^{1} = k_{1}, …, S_{n}^{d} = k_{d}) = \binom{n}{k_{1}, …, k_{d}}p_{1}^{k_{1}} ⋯ p_{d}^{k_{d}},

where (k_{1}, …, k_{d}) are nonnegative integers summing up to n. Clearly,

S_{n}^{1} + ⋯ S_{n}^{d} = n,

so the random variable S_{n}^{1}, …, S_{n}^{d} cannot be independent. The question is:

Suppose that the number of balls is itself a random variable, independent of everything else. Is there a way to choose the law of this random variable so that the above variables are independent?

To put the problem in precise mathematical terms, let ξ_{1}, ξ_{2}, … be i.i.d. random colours, i.e. random variables taking values in {c_{1}, …, c_{d}} such that

P(ξ_{1} = c_{i}) = p_{i}, 1 ≤ i ≤ d.

Let

S_{n}^{i} := ∑_{k=1}^{n}1(ξ_{k} = c_{i}).

(In physical terms, S_{n}^{i} denotes precisely what we talked about earlier using a more flowery language.) Now, independent of the sequence ξ_{1}, ξ_{2}, …, let N be a random variable with values in ℤ_{+}. The problem is to find its law so that

S_{N}^{1}, …, S_{N}^{d} are independent random variables. (⋆)

It turns out that this is a characterising property of the Poisson law. We will contend ourselves by proving one direction:

Lemma 5.5. If N is Poisson then (⋆) holds. Moreover, if N has expectation λ, then S_{N}^{i} is also Poisson with expectation λp_{i}.

EXERCISE 4. Prove the last lemma.

5.3 Geometric

A random variable X with values in ℕ is geometric if it has the memoryless property:

For each k ∈ ℕ, the conditional distribution of X − k given {X > k} is the same as the distribution of X:

P(X − k = n|X > k) = P(X = n).

Think of X as a random time, e.g. the day on which a certain volcano will erupt. The property above says that if by day k the volcano has not erupted then the remaining time X − k has the same law as X, no matter how large k is.

Lemma 5.6. If q = P(X > 1) then

P(X > k) = q^{k}, k ∈ ℕ

and

P(X = k) = pq^{k−1},

where p = 1 − q.

Proof By the property of X,

P(X > k + n|X > k) = P(X > n),

for all k, n, which means that

P(X > k + 1) = P(X > k)P(X > 1).

Iterating this, we find

P(X > k) = P(X > 1)^{k}, k = 0, 1, ….
□

People refer to X as geometric with parameter p. The terminology is not standard because other people refer to X − 1 (which also has the memoryless property but takes values in ℤ_{+}) as geometric with parameter q. A matter of taste, really. It is easy to see that Lemma 5.7. If X is geometric in ℕ with P(X = 1) = p then

EX = 1/p, var X = (1 − p)/p^{2}, Ee^{θX} = \frac{pe^{θ}}{1 − (1 − p)e^{θ}} .

EXERCISE 5. Do all that.

A concrete way to get a geometric random variable is by considering ξ_{1}, ξ_{2}, … to be i.i.d. with

P(ξ_{1} = 1) = 1 − P(ξ_{1} = 0) = p

and by letting

X = inf{k ≥ 1 : ξ_{k} = 1}.

We have P(X < ∞) = 1, so

X = min{k ≥ 1 : ξ_{k} = 1}

and

P(X > k) = P(ξ_{1} = ⋯ = ξ_{k} = 0) = (1 − p)^{k},

as required. We have that

Lemma 5.8. If X_{1}, X_{2}, …, X_{d} are independent and geometric then X = min(X_{1}, …, X_{d}) is geometric.

Proof

P(X > k) = P(X_{1} > k, …, X_{d} > k)
= P(X_{1} > k) ⋯ P(X_{d} > k)
= q_{1}^{k} ⋯ q_{d}^{k} = (q_{1} ⋯ q_{d})^{k}.
□

EXERCISE 6. Let X, Y be independent and geometric. Show that

P(X − Y > n|X > Y) = P(X > n),

for all n, and interpret the formula.

5.4 Uniform

We have already seen, in detail, how to construct a uniform random variable, from first principles. Recall that U is uniform in the interval [0, 1] if P(U ≤ x) = x, 0 ≤ x ≤ 1. More generally,

X is uniform in [a, b] if, for all intervals I, the probability P(X ∈ I) is proportional to the length of I.

Of course, if X is uniform in [a, b], then cX + d is uniform in the interval with endpoints ca + d and cb + d. Recall that if F is a distribution function and U is uniform in [0, 1] then F^{−1}(U) is a random variable with distribution function F.

Lemma 5.9. Let p_{1}, …, p_{d} be positive numbers adding up to 1. Split the interval [0, 1] into consecutive intervals I_{1}, …, I_{d} of lengths p_{1}, …, p_{d}, respectively. Let U_{1}, …, U_{n} be i.i.d. uniform in [0, 1]. Let

S_{n}^{i} = ∑_{k=1}^{n}1(U_{k} ∈ I_{i}), 1 ≤ i ≤ d.

Then (S_{n}^{1}, …, S_{n}^{d}) has a multinomial law. In particular, S_{n}^{i} is Binomial with parameters n, p_{i}.

EXERCISE 7. Show this last lemma.

EXERCISE 8. Let U_{1}, …, U_{d} be i.i.d. uniform in [0, 1]. Compute the probability P(U_{1} < U_{2} < ⋯ < U_{d}).

EXERCISE 9. Consider a stick of length 1 and break it into 3 pieces, by choosing the two break points at random. Find the probability that the 3 smaller sticks can be joined to form a triangle.

EXERCISE 10. Pick a random variable U_{1} uniform in [0, 1]. Let U_{2} be the midpoint of the interval [0, U_{1}] or of [U_{1}, 1], with equal probability. Continue in the same manner and define U_{3} to be the midpoint of [0, U_{2}] or of [U_{2}, 1], with equal probability. Show that the x ↦ lim_{n→∞} P(U_{n} ≤ x) is continuous but not absolutely so.

5.5 Exponential

A random variable T with values in ℝ_{+} is exponential if it has the memoryless property:

For all t, s > 0,
P(T − t > s|T > t) = P(T > s).

Lemma 5.10. If T is exponential then there is λ > 0 such that

P(T > t) = e^{−λt}, t ≥ 0.

Proof Implicit in the definition is that P(T > t) > 0 for all t. Hence α := P(T > 1) ∈ (0, 1). We have

P(T > t + s) = P(T > t)P(T > s)

for all t, s. Using this and induction, we have that, for all n ∈ ℕ,

P(T > nt) = P(T > t)^{n}.

This gives that, for all m ∈ ℕ,

P(T > 1 = m(1/m)) = P(T > 1/m)^{m}

and so P(T > 1/m) = P(T > 1)^{1/m}. Letting t = 1/m in the pre-last display, we have

P(T > n/m) = α^{n/m}.

Now, for t > 0, let q_{1} > q_{2} > … be rational numbers with inf{q_{1}, q_{2}, …} = t. Then

P(T > t) = P(∪_{k}{T > q_{k}}) = sup_{k} P(T > q_{k}) = sup_{k} α^{q_{k}} = α^{inf_{k} q_{k}} = α^{t}.

Since α < 1, we have λ := − log α > 0. □

We say that T is exponential with parameter (rate) λ. It is easy to see that Lemma 5.11. If T is exponential with rate λ then T has density

f(t) = λe^{−λt}, t ≥ 0.

Also,

Ee^{θT} = \frac{λ}{λ − θ} ,

defined for all θ < λ, and

ET = 1/λ, var T = 1/λ^{2}.

Proof Note that

∫_{0}^{t} f(s)ds = 1 − e^{−λt},

showing that f is a density for T. The rest are trivial. □

Lemma 5.12. Let T_{1}, T_{2}, …, T_{d} be independent exponential random variables with parameters λ_{1}, λ_{2}, …, λ_{d}, respectively. Then min(T_{1}, …, T_{d}) is exponential with parameter λ_{1} + ⋯ + λ_{d}.

Proof

P(min(T_{1}, …, T_{d}) > t) = P(T_{1} > t) ⋯ P(T_{d} > t).
□

Whereas an exponential is the natural analogue of a geometric, in that they are both memoryless, the former also enjoys the important scaling property:

If T is exponential with rate λ then, for any c > 0, cT is exponential with rate λ/c,

and this is obvious.

EXERCISE 11. Let T_{1}, T_{2}, …, T_{d} be independent exponential random variables all with rate 1. Show that

law of max(T_{1}, …, T_{d}) = law of (T_{1} + \frac{T_{2}}{2} + ⋯ + \frac{T_{d}}{d} ) .

Another relation between geometric and exponential is the following: Let p be very small. Let X be geometric with parameter p. Consider a scaling of X by p, i.e. the random variable pX which takes values p, 2p, 3p, . . .. Then the law of pX converges to an exponential law with rate 1:

Lemma 5.13 (Rényi). For X geometric with parameter p,

lim_{p→0} P(pX > t) = e^{−t}, t > 0.

Proof P(pX > t) = P(X > t/p), and, since X is an integer,

P(X > t/p) = P(X ≥ ⌈t/p⌉) = (1 − p)^{⌈t/p⌉+1}.

Since lim_{n→∞}(1 + n^{−1})^{n} = e, we have that the last expression converges to e^{−t} as p → 0. □

5.6 Gaussian (normal) variables

5.6.1 The Gaussian law

The motivation of the Gaussian probability comes from the central limit theorem (which, long time ago, was known as the “law of errors”). This was stated, without proof, in 2.3.2 . We work heuristically in order to motivate the definitions. Let S_{n} = ξ_{1} + ⋯ + ξ_{n} be the sum of n independent indicator (also known as Bernoulli) random variables ξ_{i} with P(ξ_{i} = 1) = p for all i. Let Ŝ_{n} = S_{n} − ES_{n}. Then, as n → ∞, the distribution function of Ŝ_{n}/\sqrt{n} converges to an absolutely continuous distribution function with a famous formula. Let X be a random variable with such a distribution function. Note that

\frac{Ŝ_{2n}}{\sqrt{n}} = \frac{1}{\sqrt{2}} \frac{Ŝ_{n}}{\sqrt{n}} + \frac{1}{\sqrt{2}} \frac{Ŝ′_{n}}{\sqrt{n}},

where Ŝ′_{n} = ξ_{n+1} + ⋯ + ξ_{2n} is a random variable with the same law as Ŝ_{n}. So the distribution of \frac{Ŝ′_{n}}{\sqrt{n}} will also converge to the distribution of X. Moreover, Ŝ′_{n}, Ŝ_{n} are independent. Therefore, if X_{1}, X_{2} are independent random variables with the same law as X, then we expect that

X = \frac{X_{1} + X_{2}}{\sqrt{2}} .

Even if we do not know what this famous distribution is, it should be such that this “addition law” holds. From this, we can discover its density. One way to do that is by imposing the extra assumption (which is not necessary) that the generating function of X exists for all θ: M(θ) = Ee^{θX}. Then

M(θ) = M(θ/\sqrt{2})^{2}.

Letting θ = \sqrt{η} and taking logarithms, we have

log M(\sqrt{η}) = 2 log M(\sqrt{η/2}).

So, if we temporarily let m(η) = log M(\sqrt{(}η)) we have

m(η) = 2m(η/2).

So m(0) = 0 and with some work, we can actually find that the only continuous function satisfying the latter is linear: m(η) = cη. (This should be geometrically obvious.) Hence

M(θ) = e^{cθ^{2}}.

We know that the moments of X are given by the derivatives of M at 0. We find

M′(0) = 0, M′′(0) = 2c.

So EX = 0 (as it should), while EX^{2} = 2c. So c > 0. Since EX = 0, the second moment is the variance and we customarily denote it by σ^{2}. We arrive at

M(θ) = e^{\frac{1}{2} σ^{2}θ^{2}}.

We know that there is only one probability distribution with a given moment generating function. Instead of trying to figure out which one it is, let us reveal the result and then just verify that it is correct. We claim that the probability distribution corresponding to the last M is absolutely continuous with density

f(x) = Ce^{−x^{2}/2σ^{2}}.

Here, C is such that ∫_{ℝ} f(x)dx = 1. This is the famous normal or Gaussian density with mean 0 and variance σ^{2}. It is called standard if σ^{2} = 1.

EXERCISE 12. Show that ∫_{−∞}^{∞} e^{θx}f(x)dx = e^{\frac{1}{2} σ^{2}θ^{2}}. (Hint: Complete the square and use the definition of C.)

Now, to find C is a very important thing. It is based on the following:

Lemma 5.14.

∫_{ℝ} e^{−x^{2}}dx = \sqrt{π}.

(Liouville said that a Mathematician is someone for whom this integral is obvious.)

EXERCISE 13. Use Fubini’s theorem to write (∫_{ℝ} e^{−x^{2}}dx)^{2} = ∫_{ℝ^{2}} e^{−x^{2}−y^{2}}dxdy and do the latter integral using polar coördinates.

Using this we find that

C = \frac{1}{\sqrt{2πσ^{2}}} .

Therefore, the standard normal density is

φ(x) = \frac{1}{\sqrt{2π}} e^{−x^{2}/2}.

We write N (0, 1) for the law of a random variable X with standard normal density. We write N (μ, σ^{2}) for the law of σX + μ.

EXERCISE 14. Show that a density for N (μ, σ^{2}) is

\frac{1}{\sqrt{2πσ^{2}}} exp \{−\frac{(x − μ)^{2}}{2σ^{2}} \} .

EXERCISE 15. Show that if X_{i}, i = 1, …, d are independent and X_{i} having law N (μ_{i}, σ_{i}^{2}) then ∑_{i=1}^{d} X_{i} has law N (∑_{i} μ_{i}, ∑_{i} σ_{i}^{2}). Therefore linear combinations of independent normal variables are normal.

5.6.2 The multidimensional Gaussian random vector

We now pass on to defining a Gaussian (or normal) random variable in ℝ^{d}.

We say that (X_{1}, …, X_{d}) is Gaussian in ℝ^{d} if, for all a_{1}, …, a_{d} ∈ ℝ, the random
variable a_{1}X_{1} + ⋯ + a_{d}X_{d} is normal.

The next lemma shows what the moment generating function of a normal vector is:

Lemma 5.15. If (X_{1}, …, X_{d}) is Gaussian vector with

μ_{j} = EX_{j}, r_{jk} = cov(X_{j}, X_{k}),

then

E exp ∑_{j=1}^{d} θ_{j}X_{j} = exp \{∑_{j=1}^{d} μ_{j}θ_{j} + \frac{1}{2} ∑_{j=1}^{d} ∑_{k=1}^{d} r_{jk}θ_{j}θ_{k}\} .

Proof By definition ∑_{j=1}^{d} θ_{j}X_{j} should be normal, i.e. have a law N (μ, σ^{2}), for some μ, σ^{2}. We have

μ = E ∑_{j=1}^{d} θ_{j}X_{j} = ∑_{j=1}^{d} μ_{j}θ_{j}, σ^{2} = cov ∑_{j=1}^{d} θ_{j}X_{j} = ∑_{j=1}^{d} ∑_{k=1}^{d} θ_{j}θ_{k} cov(X_{j}, X_{k}).

□

Since the moment generating function of a Gaussian vector is the exponential of a quadratic form, there is no better way to express it other than using Linear Algebra. To this end, we think of the elements x of ℝ^{d} as column vectors. We use x′ to denote transposition, i.e. the corresponding row when x is a column. And, of course, (x′)′ = x. Consider the mean (column) vector

μ = (μ_{1}, …, μ_{d})′

and the symmetric covariance matrix

R = [r_{jk}].

Write also X for the column with entries X_{1}, …, X_{d}. We then have

Ee^{θ′X} = exp \{θ′μ + \frac{1}{2}θ′Rθ\} .

This uniquely defines the law of (X_{1}, …, X_{d}). This law is denoted by N (μ, R), where μ is the mean vector and R the covariance matrix.

Lemma 5.16. A Gaussian vector (X_{1}, …, X_{d}) is absolutely continuous if and only if R is invertible. In this case, its density is given by

f(x) = \frac{1}{\sqrt{(2π)^{d} det(R)}} exp (−\frac{1}{2}(x − μ)′R^{−1}(x − μ)) .

Proof Assume μ = 0, to ease notation. Assume R is invertible. It is easily seen that

R = E(XX′)

where XX′ is a d × d matrix and E(XX′) is the matrix formed by taking the expectations of the entries of XX′. This R has two important properties:

(i) it is symmetric (obviously); (ii) it is positive semi-definite, i.e. the quadratic form

u′Ru = ∑_{k} ∑_{ℓ} R_{kℓ}u_{k}u_{ℓ} ≥ 0

for all values of the variables, positive or negative. The reason for the latter is that u′Ru is the expectation of a non-negative quantity:

u′Ru = u′EXX′u = E(X′u)′(X′u) = E(X′u)^{2} ≥ 0.

Standard linear algebra shows that R has exactly d (counting multiplicities) non-negative eigenvalues. (In fact, they are strictly positive, due to the invertibility of R which is tantamount to det(R) ≠ 0). Furthermore, the eigenvectors can be chosen to be orthonormal. Letting U be the matrix whose columns are these d orthonormal eigenvectors and Λ the diagonal matrix with the eigenvalues in its diagonal, we have

RU = UΛ,

from the very definition of the eigenvectors. Now

U′ = U^{−1},

hence

R = UΛU′ = UΛ^{1/2}Λ^{1/2}U′ = P P′,

with

P = UΛ^{1/2}.

The matrix P is non-singular and is called the square root of R. We now define new random variables Z by

X = P Z .

The thing to observe is that the covariance matrix of Z is

cov(Z) = EZZ′ = EP^{−1}XX′P^{′−1} = P^{−1}R′P^{′−1} = P^{−1}P P′P^{′−1} = I,

I being the identity matrix. Thus

Ee^{θ′Z} = e^{θ′θ/2} = ∏_{j=1}^{d} e^{θ_{j}^{2}/2} = ∏_{j=1}^{d} Ee^{θ_{j}Z_{j}},

implying that the components of Z are independent standard normal. Hence the density g of Z is product:

g(z) = ∏_{j} \frac{1}{\sqrt{2π}} e^{−z_{j}^{2}/2}.

Now X = P Z, hence its density f is computed easily by

f(x) = g(P^{−1}x)/| det(P)|,

which yields the desired formula. □

If R has determinant zero then it is possible to “reduce the dimension” of the random vector X so that the density exists. In fact,

Lemma 5.17. The support of X is the range of its covariance matrix R.

Proof Suppose that R has rank r. Then it has d − r eigenvalues at zero, so that the matrix Λ consists of a d − r size block of zeros and the remaining non-zero eigenvalues. Hence now R = P P′, where P is a d × r matrix with rank r. We try again to find Z so that

X = P Z,

where Z is an r-dimensional random vector with density. If we manage to do this we will finish, since the range of P is the range of R. Observing that P′P is an r × r non-singular matrix, we pre-multiply by it to get P′P Z = P′X, hence Z = (P′P)^{−1}P′X. So if we define Z this way, we see that

EZZ′ = (P′P)^{−1}P′RP (P′P)^{−1} = (P′P)^{−1}P′P P′P (P′P)^{−1} = I,

i.e. Z is a collection of r independent standard normal variables. The formula for Z is the formula that solves a full-rank over-estimated linear system. We usually write Z = P^{+}X and call P^{+} the pseudo-inverse of P. It remains to show that every linear function of X is a linear function of Z. Let

F = λ′Xa , G = λ′P Z

with Z = P^{+}X. Consider

F − G = λ′(X − P Z).

Then

E(F − G)^{2} = Eλ′(X − P Z)(X′ − Z′P′)λ = Eλ′(XX′ − XZ′P′ − P ZX′ + P ZZ′P′)λ.

But

EZX′ = (P′P)^{−1}P′EXX′ = (P′P)^{−1}P′P P′ = P′.

So

E(F − G)^{2} = λ′(P P′ − P P′ − P P′ + P P′)λ = 0.
□

Terminology: If X has law N (0, R) with R having rank r then ∑_{j} X_{j}^{2} is called χ^{2} with r degrees of freedom.

5.6.3 Conditional Gaussian law

If it appears that we’ve done a lot of Linear Algebra, then this is because it is so: Dealing with Gaussian random variables (and processes!) is mostly dealing with Linear Algebra (or Linear Analysis!). Without proof, we mention the following:

Lemma 5.18. Let (X; Y_{1}, …, Y_{d}) be a Gaussian random variable in ℝ^{1+d}. Then the conditional law P(X ∈ ·|Y_{1}, …, Y_{d}) is normal with mean E(X|Y_{1}, …, Y_{d}) and deterministic covariance matrix. Moreover, E(X|Y_{1}, …, Y_{d}) is a linear function of Y_{1}, …, Y_{d}) and is characterised by the fact that X − E(X|Y_{1}, …, Y_{d}) is independent of (Y_{1}, …, Y_{d}).

EXERCISE 16. Let X, Y be independent standard Gaussian variables. Based on the last part of the lemma above, compute E(X + 2Y |3X − 4Y).